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Sir: 
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IN THE CLAIMS: 

Cancel Claims 10, 15, 20, 24, 37 and 4L 
Add Claims 44-51. 

Amend Claims 1, 7, 9, 21, 25, 26, 29, 31-33, 40, and 42 as follows: 

1. (Amended) A computer-based learning system for providing an interactive 
lesson between the computer and a student user comprising: 

user input means including at least audio input means for delivering audible user 
responses to said system and speech recognition means associated with said audio input 
means; 

user interface means including at least audio output means; 

program controller means for dynamically generating said interactive lesson; and 

a plurality of databases for access by said program controller means in interpreting 

user responses input to said user input means including at least one lesson database and at 

least one lesson-based speech interpretation database. 

Claim 7, line 10, after "program controller means for" insert —dynamically—. 
Claim 9, line 3, delete "storage". 
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21, A computer-based learning systeni for providing an interactive lesson 
between the computer and a student user comprising: 

user input means including at le^t audio input means for delivering audible user 
responses to said system and speech recognition means associated with said audio input 
means; 

user interface means including at least audio output means; 
program controller means for dynamically generating said interactive lesson; and 
a plurality of databases for access by said program controller means in interpreting 
user respo nses input to said user input means including at least one lesson database, a 
plurality of Iesson»based speech interpretation databases at least comprising one database 
of anticipated incorrect student responses and one database comprising the set of all 
possible combinations of correct student responses; and at least one database of acoustic 
information for use by said speech recognition means in interpreting student responses. 

25. (Amended) A system for recognizing speaker uttermices of known textual 
material comprising: 

means for receiving audible input from said speaker; 
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at least one database for storing a plurality of versions of said known textual 
material including at least a database of all possible combinations of correct student 
responses: 

speech recognition means for comparing said audible input to said plurality of 
versions for generating an output based upon said comparing. 

Clmm 26, line 1, delete "at least one database" and insert -said plurality of 
databases—. 

29. (Amended) A system for providing effective speech recognition bv a speech 
recognition system of user utterances of text comprising: 
means for inputting text to said system; 

means for generating a plurality of phorteme sequences based on said input text; 
means for generating a plurality of variations of said input text; and 
at least one storage means for storing said plurality of phoneme sequences and said 
plurality of variations of smd input text for use bv said speech recognition system . 

3 1 (Amended) A method for providing an interactive language lesson between 
a student user and a computer having at least a program controller, speech recognition 
means, at least one first database for storing said lesson and at least one second database 
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for storing iesson-based speech interpretation information, user input means for providing 
responses from said student user, and output means for providing information relating to 
said lesson to said user, comprising program controller steps of: 

determining an interaction level for said student user: and 

dynamically generating said interactive language lesson by: 

retrieving lesson information appropriate to said interaction level from said at least 
one first database and lesson-based speech interpretation information corresponding to 
smd lesson information from said at least one second database; 

providing lesson information to said user at said output means; 

providing lesson-based speech recognition information to said speech recognition 

means; 

prompting said student user to provide responses at said user input means; 
receiving student responses from said user input means; 
interpreting said student responses at said speech recognition means; and 
generating an output based upon said interpreting of said student responses. 

32. (Amended) The method of Claim 31, wherein said computer has at least one 
additional database for storing student level information, and wherein said determining 
comprises [fiuther comprising] the step of retrieving student level mformation from said at 
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least one additional database; and adjusting said lesson information based upon said 
retrieval of student level information. 

Claim 33, line 2, after "on" insert -said interpreting of-. 

Claim 39, line 1, change "system" to --method*-. 

40. (Amended) A method for speech recognition means to interpret student 
utterances of known text in a lesson comprising: 
receiving student utterances; 

accessing lesson-based speech interpretation information from one or more 
lesson-based database and acoustic information from at least one acoustic database: 

comparing said accessed information to said [receiving] regdved utterances; and 
generating an output based upon said comparing, 

42. (Amended) A method of providing effective speech recognition bv a speech 
recognition system of user utterances of text comprising: 
inputting text to said system; 

generating a plurality of phoneme sequences based on said input text; 
generating a plurality of variations of said input text; and 
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storing said plurality of phoneme sequences and said plurality of variations of said 
input text for use bv said speech recognition system . 

44. (Added) The system of Claim I wherein said plurality of databases includes a 
reading level information database for use by the program controller in adjusting the 
complexity of the interactive lesson. 

45. (Added) The system of Claim 6 wherein said plurality of databases includes at 
least one story text database including story text and phrasing indicators for use by the 
program controller in presenting the interactive lesson. 

46. (Added) The system of Claim 6 wherein said plurality of databases includes at 
least one story page layout database for use by the program controller in presenting said 
interactive lesson, 

47. (Added) The system of Claim 1 wherein said plurality of databases 
mcludes a session database for maintaining user session information. 

48. (Added) The system of Claim 47 wherein said program controller is adapted 
to replay segments of a user session based on input from said session database. 
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49. (Added) The system of Claim 47 wherein said program controller receives 
input from said session database for establishing a starting point for said interactive lesson 
based on said user session information, 

50. (Added) The system of Claim 6 wherein said plurality of databases further 
comprises at least one display prompt database for use in generating audio output for user 
selected portions of said visual output. 

5 1 . (Added) The system of Claim 1 vdierein said plurality of databases includes at 
least one text power set database for use by said speech recognition means for interpreting 
user input. 

REMARKS 

Claims 1-43 were pending in the parent application, and Claims 10, 15, 20, 24 and 
37, which are allowed in the parent application, have been cancelled by this Preliminary 
Amendment. This amendment responds to the new rejections raised in the Final OflBce 
Action dated June 21, 1999 for the parent application. 
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In the Final Office Action, Claims 1-9, 11-14, 16-19, 21-23, 25-36 and 38-43 
stand rejected under 35 USC §102(e) as anticipated by Mostow, et al; Claims 11-12 are 
rejected as obvious in view of Mostow, et al; and Claims 21-23, 25-26, and 28-30 are 
rgected under 35 USC §103 as unpatentable in view of the combined teachings of 
Mostow and Shpiro; and Claims 31-36 and 38-40 as anticipated by Waters. 

The Examiner has rejected Claims 1-9, 11-14, 16-19, 21-23 and 38-43 under 35 
USC § 102(b) ^ven the teachings of the Mostow, et al reference. Applicants r^pectfully 
submit that the Mostow reference does not anticipate the invention as claimed. The 
Mostow reference provides for a reading coach that "listens" to student utterances and, if 
necessary, asks the student to repeat the suspect word and then speaks the correct word 
(see: Page 78, col. 1, line 17). The Mostow article specifically states that the system 
"...just has to detect where the reader is and which words were missed~it doesn't have to 
identify what the speaker said instead" (see: Page 77, Col. 2, lines 32-35). Moreover, the 
Mostow article defines "...mistakes as 'unportant words that the reader failed to ^ak'" 
(Page 77, Col. 2, lines 19-20) and goes on to state that "mispronunciations... are not 
considered reading mistakes and must therefore be tolerated" (Page 77, Col. 1, lines 
39-41). Therefore, it is clear that the Mostow system comprises a speech tracking system 
rather than a speech recognition system (Claims 1, 7, 21, 25, 29, 31, 40, 42) with speech 
recognition means and a program controller having access to at least one lesson-based 
speech interpretation database (Claims 1, 21 and 31); a plurality of phoneme sequences 
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and variations of said input text (Claims 29 and 30); a database of anticipated incorrect 
student responses (Claims 2 and 26); a database of all possible combinations of correct 
student responses (Claims 3 and 27); and/or at least one database of acoustic information 
for use by the speech recognition means in inteipreting student responses (Clmms 4, 5, 30, 
41 and 43). 

To anticipate an invention, a reference must teach each and every claimed feature. 
In light of the fact that the Mostow article does not teach a speech recognition system 
(Claims 1, 7, 21, 25, 29, 31, 40, 42) with speech recognition means and a program 
controller having access to at least one lesson-based speech inteipretation database 
(Claims 1, 21 and 31); a plurality of phoneme sequences and variations of smd input text 
(Clmms 29 and 30); a database of anticipated incorrect student responses (Clmms 2 and 

26) ; a database of all possible combinations of correct student responses (Claims 3 and 

27) ; and/or at least one database of acoustic information for use by the speech recognition 
means in interpreting student responses (Clmms 4, 5, 30, 41 and 43), Applicants contend 
that an anticipation rejection cannot be maintained. Since the limitations of the 
indept^ident Clahns, 1, 7, 21, 25, 3 1 and 40 are clearly not taught by the Mostow article, 
neither those claims nor the claims which depend therefrom are anticipated by Mostow. It 
is fiirther maintained that the clmm features recited in those independent claims are not 
obviated by the Mostow article since Mostow neither teaches nor suggests actual speech 
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recognition, any lesson-based int^retation or any mistake detection other than a reader's 
Mure to speak a word. 

With respect to the rejection of Claims 11 and 12 under 35 USC §103, in view of 
the teachings of the Mostow reference^ Applicants respectfiilly submit that, although the 
Mostow system does monitor where the student is in the text (i,e., tracks the student), and 
discusses that resuhs should be evaluated, there are no teachings in the Mostow article 
which provide one skilled in the art with sufficient information to enable them to 
hnplement an automatic monitoring or evaluatmg function in combination with speech 
recognition. Since the Mostow system does not recognize speech and is not equipped to 
recognize or mterpret any mistakes other than a readers failure to speak a word, it cannot 
be maintained that the Mostow system monitors student progress based on responses 
(Claim 11, and also Claims 16^ 23 and 33) nor that the Mostow system generates student 
progress mformation and stores same (Claim 12, and Claims 17 and 34), 

In response to the Examiner's rejection of Claims 21-23, 25-26 and 28-30 based on 
a combination of the Mostow and Shpiro patent teachings. Applicants reiterate that the 
Mostow patent does nbt provide any speech recogtiition means (as set forth in 
independent Claims 21, 25) nor any lesson storage database for storing an interactively 
generated lesson (Clahn 22), nor monitoring student progress, generating information 
regarding the student's progress, nor altering the level of interaction with the us^ based 
on the monitoring (Clmm 23), nor at least one database of anticipated mcorrect responses 
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(Claim 26). Further, the Shpiro patent does not disclose those teachings which are 
missing from the Mostow article. As set forth in arguments submitted m the parent patent 
application, the Shpiro system simply retrieves and presents a statically stored lesson. The 
Shpiro patent does not disclose any lesson storage database for storing an interactively 
generated lesson (Claim 22) nor does it teach momtoring and altering the level of 
mteraction based on the monitoring (Claim 23% nor does it disclose the use of additionally 
stored acoustic information for speech interpretation (Claim 28). While Shpiro does 
include a database having one correct and several incorrect versions of the lesson text, 
such is not the same as teaching or ^ggesting the database of anticipated incorrect 
responses of Claim 26 in combination with the additionally cited limitations of Claim 25 
from which it depends. 

Similarly, Claims 29 and 30 are not obviated by the combination of teachings of 
Mostow and Shpko. Claim 29 recites a means for inputting text to a system, means for 
generating a plurality of phoneme sequences of input text, along with means for generating 
a plurality of variations of the input text, and means for storing the generated information. 
Neither the Mostow nor the Shpiro patent teaches these automatic means for generating 
database mformation for later use in speech interpretation. Since ndther patent teaches or 
suggests such means, it is respectfully maintained that the apparatus Claims 29-30 as well 
as the parallel method Claims 42 and 43 are allowable over the cited art. 
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Finally, the Examinet had rgected Clmms 31-36 and Clmms 38-40 as anticipated 
by the Waters patent. The Waters patent provides a language lesson system which 
compares student input to the "course of study**, generates audio output and which fiirther 
"varies the course... in dependence on the user*s temporal history of correct and incorrect 
replies" (see: column 4, lines 40-47), It is respectMy submitted that the Waters system 
does not anticipate the claimed invention since the Waters system does not provide 
lesson-based speech recognition information to the speech recognition means. Rather, the 
Waters system interprets the student input and then compares it to the lesson to determine 
if the student input is correct or incorrect. The Waters system does not use the 
lesson-based speech r^ognition information in recognizing the student input (mdependent 
Clmm 31 and Claims 32-36 and 38-39 which depend therefrom). Further, the Waters 
system simply keeps track of the "temporal history** of correct and incorrect replies, which 
does not anticipate the claimed monitoring of progress based on intei^retation of student 
responses (Claim 33 and Claims 34-36 which depend therefrom). As such. Applicants 
believe that the Waters patent does not anticipate the invention as set forth in Claims 
31^36 and 38-39. 

While the Examiner has cited the Waters patent against Claim 40 as well. 
Applicants believe that the previously-submitted amendments, which are reiterated in this 
Preliminary Amendment to the continuation application, successfiiliy distinguish Claim 40 
from the teachings of the Waters patent. The Waters patent does not provide an acoustic 
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interpretation database, nor does it interpret student responses using any input that is 
iesson-based. ITierefore, Applicants assert that the Waters patent does not anticipate 
Claim 40. 

In light of the foregoing, it is respectfully requested that the objections and 
rejections to this Application be reconsidered and withdrawn, the newly submitted claims 
be entered, and the remaining Claims 1-9, 11-14, 16-19, 21-23, 25-36, 38-40 and 42-51 
be passed to issuance. 



Respectfijlly submitted. 



H. W. Adams, et al 




Anne Vachon DougHert] 
Registration No. 30,374 
Tel. (914) 962-5910 
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V SYSTEM AND METHOD FOR INTERACTIVE READING AND LANGUAGE 

INSTRUCTION 



Fjeld of the Invention 



This invention relates to computer-based instruction and more 
particularly to the use of computers having speech recognition 
software uniquely provided to simulate continuous speech 
recognition of juvenile speech in order to teach reading and/or 
language. The system further provides an interactive, variably- 
paced course of instruction, the progress of which can be 
automatically adjusted, and provides for uploading or display of 
the lesson workproduct at any time during or after completion of a 
lesson. 



Computer instruction for reading and language was first 
implemented in the 1970s and has progressed in complexity with 
developments in the technology* An early system, as detailed in 
U.S. Patent No. 4,078,319 Of Mazesli, et al, utilized optical 
components to train the student's eyes to focus on particular 
material and to advance laterally along the line of material to be 
taught, thereby teaching the mechanics of reading. Increasingly 
sophisticated systems incorporated audio presentation of the 
y0996-244 ^ 1 
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material to be learned, thereby simulating a human instructor and 
providing a degree of passive error control and correction. 
Systems which provide audio presentation are particularly well 
suited for foreign language instruction, 

h reading teaching system having simultaneous audio and visual 
display of material is taught in U.S. Patent No. 4,397,635 of 
Samuels, wherein the audio and/or the visual display of the 
material is altered to enhance the student's understanding of the 
meaning of the material to be learned (e.g*, lower volume audio and 
downwardly-^sianted lettering would be used in presenting the word 
**down")# Yet more complex is the system described in U.S. Patent 
No* 5,540,589 Of Waters, wherein voice recognition is utilized for 
instruction and for monitoring of a student's progress. The Waters 
system is designed to repeat material which is incorrectly 
"learned,** as evidenced by recognized vocal responses, and to 
recommend termination of a session if errors exceed a preset fault 
tolerance* 

A particular challenge associated with speech recognition 
systems, and critical to successful implementation of an automatic 
interactive reading system for a young child, is development of a 
children's speech model. Due to the range of pitch of children's 
voices, the irregular and unpredictable pace at which small 
children utter sounds, syllables, words and phrases, often without 
pausing to breathe where an adult would pause, their uncertain 
articulation, and the often strong influence of foreign languages 
spoken at home, prior art speech recognition systems have been 
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unsuccessfui at effectively recognizing speech uttered by children. 
Additional challenges to speech recognition include pronunciation 
variations attributable to regional dialects, foreign accents, 
common articulation problems, and idiosyncracies associated with 
various stages of speech development. An automatic interactive 
reading/language system must be able to address the foregoing 
challenges in order to be productive. 

It is desirable to develop a system which will not only 
provide audio presentation and speech recognition, but also provide 
an interactive en^^ironment in which the tempo of presentation of 
material can be automatically adjusted to accotoodate the student's 
level of skill and advancement through the program, wherein the 
student, and particularly a juvenile student, may participate with 
the program with increasing interaction as proficiency increases. 
Xlso desirable, for purposes of independent review, entertainment, 
and reward, is a system which can display the workproduct of each 
lesson by uploading and/or local display of the interactively- 
created presentation. 

It is therefore an objective of the present invention to 
provide a computer-based reading or language instruction system and 
method, incorporating speech recognition software, for interpreting 
the student responses, interacting with the student, and altering 
the pace of the instruction based upon the student's responses. 

it is a further objective of the invention to incorporate 
means for correcting student responses and audibly generating the 
correct responses as a means for simulating the activities of a 
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teacher . 

Yet another objective of the invention is to provide a system 
incorporating a database having lessons with varying degrees of 
student interaction to automatically increasingly challenge the 
learner and means to present those lessons in varied ways depending 
upon the proficiency of the student. 

Another objective of the invention is to provide a speech 
recognition database to enhance recognition of children's speech 
and speech which is otherwise difficult to recognize. 

still another objective of the invention is to provide a 
system which will coordinate and display a joint performance of the 
inputs contributed by the student and the computer instructor in 
such a manner as to illustrate the student's reading or language 
proficiency. 

suamarv of the invention 

The above and other objectives are provided in the present 
invention which utilizes a mechanism to permit a computer-based 
instruction program in reading or language instruction to implement 
a shared reading program, wherein portions of the text are read by 
the student and portions by the computer itself. The proportion of 
the text "read" by each participant and the pace at which the 
lesson progresses can each be automatically altered based upon the 
proficiency of the student. In addition, the system represents a 
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computer companion to share the task of reading or language 
learning by modeling the task for the learner, variably controlling 
the amount of material that the student reads aloud based upon a 
computer assessment of the student's ability, helping the learner 
navigate through the instructional application, and participating 
in a joint performance that demonstrates the student's reading. 

The foregoing are accomplished by providing a system having an 
executive program, for administration and execution of the shared 
reading program; audio input means having associated speech 
recognition means with access to a plurality of databases, 
including a database built upon a unique acoustic model for 
simulating continuous recognition of speech and at least one 
lesson-based speech recognition database; optional non-speech input 
means; and lesson output means. The executive program generates a 
computer instructor, perhaps an animated character, that asks the 
student questions, diredts reading tasks, reads what the student 
does not read, and helps navigate the student through the 
application via the lesson output means, including audio and 
optional visual outputs. The system is provided with a plurality 
of databases including at least a database of textual fragments, 
organized by attributes such as difficulty and position, and means 
for defining sections for the student to read and sections for the 
computer instructor to read; and the aforementioned lesson-based 
databases of anticipated responses, including multiple variations 
of groupings of sounds in the responses and anticipated 
mispronunciations, for speech recognition. 
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The invention will now be described in further detail with 
specific reference to the attached figures wherein: 

Figure 1 illustrates a basic block diagram of the inventive 
system; 

Figure 2 schematically illustrates the system controller 
components and the intercommunications paths among same; and 

Figure 3 illustrates the textual database support for shared 
language learning. 

Figure 4 provides a representative flow chart of tasks for the 
executive program. 



The invention, in its most basic embodiment as illustrated in 
Figure 1, comprises a system controller 118, comprising an 
executive program having access to information stored in one or 
more storage locations, illustratively depicted as disk 119. It is 
to be noted that internal or external storage means other than disk 
storage could clearly be utilized, provided that the executive 
program can readily access stored data and/or applications. 
Textual or instructional input from a student user to the system 
controller is provided via one or more non-^speech user inputs. 



shown representatively as keyboard 128 or mouse 138, while audio 
input is provided via isicrophone 120. Output from the system 
controller for interaction with the student is provided as video 
output at the graphical user interface, display 107 or as audio 
output via speaker 114. Combination of components in stand-alone 
systems, or omission of one of the input or output devices, is 
common practice which may affect the necessary software available 
to the system controller, but would not alter that which is novel 
to the present system* A most basic implementation of the 
instruction system may incorporate only audio input and output 
means, with visual student stimuli being provided by a physical 
textbook or storybook. The essence of the invention concerns the 
particulars of the databases, the actions of the executive program 
and the actions of the speech engine* The other illustrated or 
mentioned components deserve no further detailed attention except 
insofar as they provide a context for clarifying the necessary 
details of the invention. Prom the student user perspective, a 
representative system would comprise a personal computer having a 
keyboard and/or a mouse for controlling the pointing device (8 of 
Figure 2), a monitor, one or more speakers and a microphone* The 
student may or may not be required to load a program from a CD--ROM 
or diskette, or download same from a dedicated network or the 
internet, depending upon the specific embodiment* 

The executive program orchestrates the instruction and 
interaction between the student user and the system, and requires 
access to additionally stored databases of relevant information. 
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representatively depicted at the storage locations on disk 119 of 
Figure 1. vmile the databases of stored information are detailed 
below, it will be understood by one having skill in the art that 
the specific contents of the databases may vary slightly without 
departing from the intent of the invention. In addition, it is 
clear that the information may be combined into one or more larger 
databases, provided that all of the information remains accessible 
in the system. The databases, with reference numerals related to 
Figure 2, are aS follows: 

reading l^vel information database 9-Gontrols information 
representing the user's recent performance and current level which 
information is used to change the amount of support given to the 
user during his or her reading; 

story text database 10-includes the text of the story and 
phrasing indicators that are used to break up the text and control 
the amount of material that the learner is asked to read; 

story page layout database 11-includes information regarding 
the number and dimensions of the pages, the positioning of text 
blocks and their sequencing, the positioning and size of graphics, 
and the coordinates used for routing the companion around the 
screen; 

session database for replay and resumption 17-contains all the 
information necessary to provide a replay of the joint reading of 
the text by the companion and the student, including the portions 
read correctly by the student, and the portions read by the 
computer instructor /companion, along with information about audio 
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annotations to the text, and an optional video introduction to the 
joint reading performance. This database also allows the student 
to resume the session at a later time, and enables storage of the 
session for uploading and transmission or for local review at a 
5 later timei 

audio clip database 16-indexes a digital representation of the 
audio for all feedback and directional messages, words, and phrases 
by their textual form; 

picture pronqpt database 18-contains pointers to audio 
l^E information maintained in the audio clip database such that by 
i student selection of a picture function, the system will generate 
jl an audio expression of what the picture represents in terms of the 
J7^ overall story; 

^ page overview database 19-contains pointers to audio 

1^ information maintained in the audio clip database such that by 
student selection of the page overview function, the system will 
0 generate an audio expression of what the contents of the page mean 
to the overall story; 

fee^t>acK message database 2l--contains all feedback messages 
20 designed to offer information to the student, categorized as to the 
type of message, such that messages within a given category are 
selected randomly from that category for delivery when appropriate; 

learner population-specific acoustic moSel 3-an acoustic 
representation of children's speech that is used by the speech 
25 recognition engine to enhance recognition of children's speech and 
assist in assigning a textual interpretation to the received speech 
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signals; 

alternate pbrase and pronunciation database 4-a phonemic 
representation of different ways to pronounce words in the 
currently active vocabulary {i.e., in the text which is being read 
5 at any given time) ; 

text power set database 12-a phonemic representation of the 
power set of contiguous words in the text which is being read tat 
any given time to effect pseudo-continuous speech recognition; 

student level c<mtroller 15-changes the level of the reading 
im task based on information contained in the reading level 
,E information database so that students receive more examples of the 
ii reading being modeled and are asked to read relatively smaller 
Q portions of the text as the level decreases, and vice versa as the 
^ level increases; and 
iC local text position management database 22-contains all 

information regarding the current point of reading which is used 
0 for managing the cursor or for determining whether a word has been 
read. 

The paths for communications and information handling among 
20 the various system components, applications and databases are 
representatively illustrated in Figure 2 and will be detailed in 
the following description of the operation of the system. When 
initiating a lesson (and throughout the lesson), the executive 
program 6 receives input from the student user via the non-speech 
25 user input interface 8 (i.e., a mouse, keyboard, touchpad, 
touchscreen, or other known non-speech input device) and/or from 
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microphone 20 via the speech recognition interface 2 and speech 
recognition engine 1. The speech recognition engine will be 
provided with information from the learner population specific 
acoustic model 3, the alternate pronunciation and phrase database 
4, and the text power set database 12 for stored information to 
facilitate recognition of the audio input. 

The learner population specific acoustic model assists the 
speech recognition engine in simulating continuous recognition of 
the learner population's, representatively children's, speech. 
Given the idiosyncracies of the speech of the anticipated users , an 
acoustic model of that speech is provided to enhance recognition. 
As noted in the Background section, the idiosyncracies of 
children's speech generally defy recognition on a continuous speech 
basis, and the acoustic model for purposes of this description is 
that of children's speech. If the anticipated user group is adult, 
adult with particular speech disorders, adult with a given foreign 
accent, teen, etc., an appropriate acoustic model can be developed 
accordingly. 

The alternate phrase and pronunciation database 4 and the text 
power set database 12 are additionally provided to enhance 
recognition of the uttered responses. Both of these latter 
databases, 4 and 12, are based upon the text of the story to be 
read. As the executive program is providing a portion of the text 
for display to the student user, or directing the student's 
attention to a particular page of text in the embodiment having no 
computer-based visual output means, the alternate phrase and 
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pronunciation versions of that text, and the power sets of that 
text are provided to the speech recognition engine in order to 
assist in interpretation of the responses uttered by the student 
user. 

The text power set database is developed based on the fact 
that certain anticipated users (and, again representatively 
children, though it may well apply to teen and adult users) may not 
breathe or pause between words or phrases, thereby rendering 
continuous speech recognition ineffective. In order to make 
discrete speech recognition effective for those anticipated users, 
power sets of contiguous words, which might be run-together, are 
developed and provided in the text power set database. 

To develop the text power set database, Knowing that it is not 
realistic to have all of the anticipated users train the speech 
recognition system, the database is generated independently and 
preferably automatically. In so doing, the signals for phoneme 
sequences in the text are input and variations of the phoneme 
sequences are generated as the contents of the database. As such, 
for example, the phrase "the dog barks" may be stored in the text 
power set database as "thedog barks" , "the dogbarks" , "thedogbarks" 
and "the dog barks". The phoneme sequences which are stored can, 
clearly, also include portions of words, separated at logical 
syllabic breaks. As might be expected, by providing many 
alternative correct responses, one is broadening the interpretation 
space around the text. As a result, many more utterances will be 
recognized as correct than would have been when using the speech 
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recognition engine alone. The alternate phrase and pronunciation 
database is, therefore, provided to assist in re-narrowing the 
search space about the actual text. 

The alternate phrase and pronunciation database provides both 
alternative correct pronunciations (e.g., "aks" for "ask") but also 
anticipated wiscues for pronunciation of the text (e.g., "dark" for 
"bark"). By accessing the alternate phrase and pronunciation 
database, the speech recognition engine can reject incorrect 
utterances which would otherwise be accepted based upon the broader 
text power sets. Automatic generation of the alternate phrase and 
pronunciation database requires training the system in linguistic 
knowledge including: common pronunciation problems, such as running 
adjacent consonant sounds together; common reading problems, such 
as seeing "was" as "saw"; common substitutions of letters, such as 
"b", "d", "p" and "q"; etc. all based upon the known text of the 
lesson . 

By iteratively building the databases based not only upon 
correct responses, but also upon variant combinations of contiguous 
correct responses and of anticipated mispronunciations influenced 
by age, foreign accent, regional dialect, etc., of the correct 
responses based on the text, the system is able to discretely 
recognize the speech as if it is being continuously recognized. 
The databases capture all anticipated variants and combinations of 
sentences, phrases, words, syllables, and sounds which could 
logically be uttered based upon the portion of the text being 
viewed for reading. Speech recognition therefore becomes a much 
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less complex matching problem when the universe of possible 
utterances is bounded by sets of anticipated responses. 

In response to student initiation of a lesson, the executive 
program will receive input from the student level controller 15 to 
determine the level of shared language lesson to be presented. The 
student level controller, 15, in turn retrieves input from the 
reading level information database, 9, concerning the user's most 
recent performance and current level of lesson. The levels of 
lesson range from a beginner level, at which all of the lesson is 
read by the computer instructor /companion, to the most advanced 
level, at which the student performs all of the reading. Levels in 
between have the computer instructor reading part of the textual 
segments, with the student reading the remaining portion(s) in 
shared reading interactions. For the sake of this description, a 
juvenile student will be assumed, and the lesson through which the 
student will be guided is the reading of an early-reader storybook. 

Once the executive program has determined an appropriate level 
for the lesson, the lesson in retrieved from associated databases, 
including the story text database 10, audio clip database 16 (if 
real-time speech synthesis is not available to the executive 
program), and story page layout database 11. At the same time, the 
executive program generates the reading companion 5 to instruct the 
student through the lesson. Text segments with page graphics are 
provided for viewing by the student at the graphical user interface 
7, with accompanying audio output provided at 14, while appropriate 
text-based information from the alternate phrase and pronunciation 
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database 4 and the text power set database 12 is provided to the 
speech recognition engine. As noted above, an alternative 
embodiment for providing instruction at a lesser cost may eliminate 
the visual display, relying upon the student's attention to a 
5 physical textbook, as cued via the audio output of the computer. 

For our early reader example, the illustrated book cover would 
be displayed along with an audio introduction. The student could 
then prompt the system, for example by clicking on a page icon, to 
open the book to begin the story. The page icon remains available 
to the student, in a continuously-displayed task bar, to turn the 
i page and proceed with the story, or even to skip pages, at any 
time. Additional icons or words may be displayed continuously for 
j7; prompting the system to perform an alternative function, as will be 
^ further described below. 
1^ For beginner and intermediate level lessons, visually 

displayed text segments will be accompanied by simultaneously 
0 rendered audio presentation of the text segments from audio output 
14, As noted above, the audio text segrments may be synthesized in 
real time by the executive program or retrieved from the audio clip 
20 database 16 and synchronized to the visual output by the executive 
program. For our early reader example, it is presumed that display 
of the text will always be accompanied by the story illustrations, 
ideally with the exact visual layout of the storybook being 
reproduced on the computer monitor, 
25 The positional pacer 17, with input from the local text 

position management databases 22, may be implemented to 
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continuously prompt the student along the text, identifying (e.g., 
"follow the bouncing ball") the word that is being read by the 
coiaputer instructor or is to be pronounced by the student. In 
addition, the executive program may highlight or color 
differentiate the text to be read by the computer and that to be 
read by the student. At each juncture of the lesson at which the 
student is to utter one or more text segments, the user's input is 
provided to the speech recognition interface 2 as described above. 
Correct input may be passively acknowledged, simply by the progress 
of the lesson to the next text segment, or actively acknowledged by 
audio output generated via access to the feedback message database 
21. Rapid progress, as sensed by the positional pacer, will cause 
the positional pacer to increase the tempo of the lesson. If the 
user correctly pronounces the "current" (e.g,, highlighted) word 
before the pacer reaches the boundary between that word and the 
next word (the right boundary for standard left- to-right English 
reading), the pacer will accelerate slightly. Similarly, if the 
pacer reaches the boundary before the user pronounces the word, the 
pacer will slow its overall rate slightly. As will be understood 
by one having skill in the art, the pacer function can be 
implemented as a separate pacer task or pacer subroutine of the 
executive program. 

Throughout the lesson, the audio inputs from both the student 
and the computer instructor, along with the text as displayed for 
utterance by each party, are stored at the session database for 
replay and resumption 17. After a lesson has been completed, the 
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student may enjoy a reprise of the lesson to review progress and to 
identify areas for concentrated effort in the future. Storage and 
optional uploading of the lesson, in edited or unedited form, can 
be provided for independent instructor, peer, parent or personal 
student review of the lesson or for remote or deferred monitoring 
of progress (e.g., in the context of a correspondence-type course 
wherein an instructor or evaluator may review the performance) . 
The aforementioned editing may refer to system incorporation of re- 
read and corrected utterances into the stored performance, and/or 
insertion of visual or audible annotations (e.g., "This was the 
easiest pagel") provided from the student by invoicing an annotation 
function. The executive program may also compile lists or annotate 
pre-compiled lists of words in the text which have been 
successfully read or missed. All of the foregoing can facilitate 
independent review of the student's progress. 

During a lesson, the student user may alternatively select 
from one or more options to review the lesson to date or to select 
"help" for audio assistance. To review the lesson, for example, 
the student would click on a "playback" icon, or otherwise input 
the appropriate trigger. In response, the executive program would 
access the session database for replay and resumption 17. As is 
evident from the name of the database, as well as from its 
described functionality, a student may return to a particular 
lesson and resume that lesson at the point at which they last 
exited the lesson, rather than having to repeat all of their 
previous work, by accessing a previous lesson stored at 17. 
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Databases IB and 19, the picture prompt database and page 
overview database, respectively, may be accessed when the student 
user selects an appropriate icon to receive audio "help." 
Specifically, in accordance with one embodiment of the invention, 
in response to the student clicking on a particular illustration or 
part of an illustration on the displayed storybook page, the 
executive program will access the picture prompt database 18 to 
generate an audio expression of what that illustration or part of 
the illustration means or represents in the context of the story. 
Similarly, it the student reader wishes to know how an entire page 
fits into the story, the student can click on an icon to retrieve 
the requested page information from the page overview database for 
audible presentation. If a student is having difficulty in 
reading/pronouncing a particular word, the student may request help 
for that one word at any time. 

As the student progresses through the lessons in the 
application, the executive program 6 regularly updates the reading 
level information database 9 with a revised estimate of the 
student's competency based upon his or her performance. Each 
update that changes the estimate can potentially change the manner 
in which the reading or speaking of a textual segment will be 
shared for the present lesson or future lessons. If the estimate 
of the student's ability improves, the interaction between the 
computer instructor and the student may move from level N (where 
all but the last N words in a text segment are "read" by the 
computer instructor) to level N+1. Each level of interaction is 
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associated with a pattern of shared initiative that is applied to 
a given text segment as illustrated in Figure 3. A feature can 
also be implemented whereby the student can select the lesson level 
prior to conmencing, or at any time during, the session. 

Figure 3 abstractly describes the layout and types of 
information maintained in the story text database 10, showing that 
each word token, depicted as 27 in text segment 28, is present 
along with a level indicator 29 which is used to determine whether 
the student is being asked to read or pronounce the token or 
whether it is being supplied by the computer instructor. Under 
this unique text storage scheme, only a single version of the text 
must be stored, from which a myriad of different renderings (i.e., 
combinations of shared reading between the instructor and the 
student user) of the text can be generated. The executive program 
interprets the level values and phrase information in the text 
database for each token to yield the interaction for a particular 
text segment. This interpretation defines exactly what is supposed 
to be read by the computer instructor and what is to be read by the 
student. Each level applies a different interpreter to the text 
segment to yield the interaction for that level. 

Figure 4 provides a representative process flow through which 
the executive program will progress in a typical lesson. At box 
201 the executive program receives student logon or session 
commencement input, followed by an optional step, at 202, of 
verifying user access to the program (this optional step is 
particularly significant for fee-based network implementations of 
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the invention and would be readily implemented in accordance with 
known logon/verification schemes). The executive program would 
next access the student level controller, at step 203, to determine 
what lesson is to be delivered at the graphical user interface, and 
5 at what level of interaction. Upon establishment of the student 
level, the executive program will access the relevant databases 
(e*g., story text, audio clip, and layout) at 204 and proceed at 
205 to generate a display of a portion of the lesson and the 
reading companion character which will audibly prompt the student 
l(h to read some or all of the displayed portion of the lesson • While 
p the text is being accessed for display at 204, the text power set 
\1 ' ^ database and alternate phrase and pronunciation databases are being 
|7j accessed for the sets of possible responses which correspond to the 
^ text to be displayed and are being provided to the speech 
iC^ recognition engine at step 239. 
!: The order in which steps 204 and 239 are conducted is 

D arbitrary, provided that the speech recognition engine has the 
information it needs prior to receipt of student utterances. The 
steps may also be implemented such that the speech recognition 
20 engine is provided with database information corresponding to the 
next expected portion of text to be provided at the user interface, 
in advance of provision of that text for display to the user. In 
that way, the speech recognition engine will be prepared in advance 
for the expected utterances. 
25 The executive program next waits to receive input from the 

student. If input is not received within a preset time frame, as 
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determined at decision box 240, the executive program may generate 
a gentle reminder to proceed, at box 241, followed by another 
decision box 242, If no response is forthcoming within another 
timed interval, the program will shutdown at 243 under the 
5 assumption that the student has abandoned the terminal. 

If, however, the answer at either decision box indicates that 
the student has responded, with the response having been received 
and recognized by the speech recognition engine having access to 
the aforementioned speech recognition enhancement databases, the 
1© executive program next evaluates the received response at box 207. 
J If the response is not correct, the executive program may access 
\2 the feedback message database and generate appropriate feedback, at 
J 211, encouraging the reader to respond again. If the repeated 
response is incorrect, as determined at box 212, the executive 
iS program may generate the correct response at 213. If a correct 
j response is received at either 207 or 212, optional positive 
fl feedback may be generated at 208. Responses received from the 
student and audio output, along with the visual output generated by 
the executive program, (collectively known as the "joint 
20 performance") are stored at the session database for replay and 
resumption in step 209. 

After storing the current segment of the lesson, the executive 
program checks to see if the previously displayed text is the last 
portion of the lesson, at decision box 214. If there is more to 
25 the lesson, the system returns to step 205. If, however, the 
lesson has been completed, the executive program will, via the 
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reading companion, inquire if the reader wishes to continue with 
another lesson, at step 215. If the student reader wishes to 
continue, the program will return to step 204 to access story text, 
etc. from the database. If the student does not wish to continue, 
the executive program may ask the user if display of the stored 
joint performance is desired, at step 217. Depending upon the 
response, the program will either display the joint performance, at 
step 219, or terminate the session, at 218. 

The program may periodically interrupt the process flow for, 
or periodically perform a parallel operation of, updating the 
reading level information database as to the student's proficiency. 
In addition, the student may randomly interrupt the process flow by 
requesting a page overview or other "help" functions as discussed 
above. The interrupts are not illustrated, but the process flow in 
response to each of the foregoing will be readily apparent to one 
having skill in the art. 

The invention has been described with reference to 
several specific embodiments. One having skill in the relevant art 
will recognize that modifications may be made without departing 
from the spirit and scope of the invention as set forth in the 
appended claims. 
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CLAIMS 



Having thus described our invention, what we claim as new and 
desire to secure by Letters Patent is; 



1 1. A computer-based learning system for providing an 

2 interactive lesson between the computer and a student user 

3 comprising: 

m user input means including at least audio input means for 

£ delivering audible user responses to said system and speech 

^ recognition means associated with said audio input means; 
C user interface means including at least audio output 

§^ means ; 

^ program controller means for generating said interactive 

Ifcf lesson; and 

itf a plurality of databases for access by said program 

12 controller means including at least one lesson database and at 

13 least one lesson-based speech interpretation database. 

1 2, The system of Claim 1 wherein said at least one lesson- 

2 based speech interpretation database comprises at least one 

3 database of anticipated incorrect student responses. 

1 3. The system of Claim 1 wherein said at least one lesson- 

2 based speech interpretation database comprises at least one 



Y0996-244 - 23 - 



3 database comprising the set of all possible combinations of 

4 correct student responses. 

1 4. The system of Claim 1 further comprising at least one 

2 database of acoustic information for use by said speech 

3 recognition means in interpreting student responses. 

1 5. The system of Claim 4 wherein said database of acoustic 

2 information comprises acoustic infoirmation relating to speech 
^ by a specific class of students. 

J 6. The system of Claim 1 further comprising visual output 

% means for display of information accessed from said at least 

*^ one lesson database by said program controller means. 

t 7. A computer-based learning system for providing an 

m interactive lesson between the computer and a student user 

3 comprising: 

4 User input means including at least audio input means for 

5 delivering audible user responses to said system and speech 

6 recognition means associated with said audio input means; 

7 user interface means including at least audio output 

8 means ; 

9 program controller means for generating said interactive 

10 lesson? and 

11 a plurality of databases for access by said program 



¥0996-244 



- 24 - 



12 controller means including at least one lesson database and at 

13 least one lesson storage database for storing the 

14 interactively generated lesson. 

1 8. The system of Claim 7 wherein said system further 

2 comprises visual output means and wherein said program 

3 controller is adapted to display said interactively generated 

4 lesson stored in said at least one lesson storage database. 

E 9, The system of Claim 7 further comprising network means 

;| for providing said interactively generated lesson stored in 

i£ said at least one lesson storage database from said computer 

Ai to a remote location. 

t 10. A computer-based learning system for providing an 

^ interactive lesson between the computer and a student user 

£ comprising: 

4 user input means including at least audio input means for 

5 delivering user speech to said system and speech recognition 

6 means associated with said audio input means; 

7 user interface means including at least audio output 

8 means and visual output means; 

9 program controller means for generating said interactive 

10 lesson; 

11 a plurality of databases for access by said program 

12 controller means including at least one lesson database; and 
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13 Wherein said program controller means additionally 

14 comprises means for providing a prompt to pace the student 

15 through said lesson and wherein said pace is determined by 

16 said executive program assessing said student speech, 

1 11. The system of Claim 1 wherein said program controller is 

2 adapted to continually monitor student progress based on said 

3 student responses. 

13 12, The system of Claim 11 wherein said program controller is 

& further adapted to generate student progress information based 

il on said monitoring; and further comprising at least one 

4^ storage location for storing said student progress 

^ information * 

i;i 13. The system of Claim 11 wherein said program controller is 

^ adapted to alter the level of interaction of the student with 

3 said lesson based upon said monitoring. 

1 14. The system of Claim 11 wherein said program controller is 

2 additionally adapted to generate at least audio output based 

3 on said monitoring. 

1 15. The system of Claim 13 wherein said program controller 

2 means additionally comprises means for providing a prompt to 

3 pace the student through said lesson and wherein said pace is 
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4 determined by said monitoring. 

1 16, The system of Claim 7 wherein said program controller is 

2 adapted to continually monitor student progress based on said 

3 student responses. 

1 17. The system of Claim 16 wherein said program controller is 

2 further adapted to generate student progress information based 

3 on said monitoring; and further comprising at least one 
m storage location for storing said student progress 
^ information* 

jj 18. The system of Claim 16 wherein said program controller is 

f adapted to alter the level of interaction of the student with 

% said lesson based upon said monitoring. 

f 19. The system of Claim 16 wherein said program controller is 

2 additionally adapted to generate at least audio output based 

3 on said monitoring. 

1 20. The system of Claim 18 wherein said program controller 

2 means additionally comprises means for providing a prompt to 

3 pace the student through said lesson and wherein said pace is 

4 determined by said monitoring. 
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1 21, A computer-based learning system for providing an 

2 interactive lesson between the computer and a student user 

3 comprising; 

4 user input aieans including at least audio input means for 

5 delivering audible user responses to said system and speech 

6 recognition means associated with said audio input means; 

7 user interface means including at least audio output 

8 means ; 

9 program controller means for generating said interactive 
ICfti lesson; and 

ll| a plurality of databases for access by said program 

1^^ controller means including at least one lesson database, a 

liCl plurality of lesson-based speech interpretation databases at 

1^ least comprising one database of anticipated incorrect student 

1^ responses and one database comprising the set of all possible 
combinations of correct student responses; and at least one 

Ivfl database of acoustic information for use by said speech 

is" recognition means in interpreting student responses* 

1 22. The system of Claim 21 further including at least one 

2 lesson storage database for storing the interactively 

3 generated lesson. 

1 23. The system of Claim 21 wherein said program controller is 

2 adapted to continually monitor student progress based on said 

3 student responses ,r to generate student progress information 

¥0996-244 - 28 - 



4 based on said monitoring; and to alter the level of 

5 interaction of the student with said lesson based upon said 

6 monitoring. 

1 24. The system of Claim 23 wherein said program controller 

2 means additionally comprises means for providing a prompt to 

3 pace the student through said lesson and wherein said pace is 

4 determined by said monitoring. 

3g 2 5 , A system for recogni z ing speaker utterances of known 

$ textual material comprising: 

means for receiving audible input from said speaker; 

at least one database for storing a plurality of versions 

# of said known textual material; 

speech recognition means for comparing said audible input 

F= to said plurality of versions and for generating an output 

® based upon said comparing. 

1 26. The system of Claim 25 wherein said at least one database 

2 comprises at least one database of anticipated incorrect 

3 student responses. 

1 27. The system of Claim 25 wherein said at least one database 

2 comprises at least one database comprising the set of all 

3 possible combinations of correct student responses. 
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1 28, The system of Claim 25 further comprising at least one 

2 database of acoustic information for use by said speech 

3 recognition means in interpreting student responses. 

1 29. A system for providing effective speech recognition of 

2 user utterances of text comprising: 

3 means for inputting text to said system; 

4 means for generating a plurality of phoneme sequences 

5 based on said input text; 

^ means for generating a plurality of variations of said 

J input text; and 

^ at least one storage means for storing said plurality of 

Wi phoneme sequences and said plurality of variations of said 

lj¥ input text. 

^ 30. The system of Claim 29 further comprising at least one 

^ database having acoustic interpretation information. 

1 31. A method for providing an interactive language lesson 

2 between a student user and a computer having at least a 

3 program controller, speech recognition means, at least one 

4 first database for storing said lesson and at least one second 

5 database for storing lesson-based speech interpretation 

6 information, user input means for providing responses from 

7 said student user, and output means for providing information 

8 relating to said lesson to said user, comprising program 
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9 controller steps of: 

10 retrieving lesson information from said at least one 

11 first database and lesson-based speech interpretation 

12 information corresponding to said lesson information from said 

13 at least one second database; 

14 providing lesson information to said user at said output 

15 means; 

16 providing lesson-abased speech recognition information to 

17 said speech recognition means; 

Ife prompting said student user to provide responses at said 

1^ user input means; 

2\t receiving student responses from said speech recognition 

2 in means; and 

2^ generating an output based upon said student responses • 

32* The method of Claim 31, wherein said computer has at 

^ least one additional database for storing student level 

3 information, further comprising the step of retrieving student 

4 level information from said at least one additional database; 

5 and adjusting said lesson information based upon said 

6 retrieval of student level information* 

1 33* The method of Claim 31 further comprising said program 

2 controller continually monitoring student progress based on 

3 said student responses* 
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1 34. The method of Claim 33 wherein said program controller is 

2 further adapted to generate student progress information based 

3 on said monitoring* 

1 35* The method of Claim 33 further comprising said program 

2 controller altering the prompting of said student based upon 

3 said monitoring. 

1 36. The method of Claim 33 further comprising said program 
^ controller generating at least audio output based on said 
£ monitoring. 

fj 37, The method of Claim 33 further comprising providing means 

T to pace the student through said lesson, and altering said 
pace based upon said monitoring* 

S 38. The method of Claim 31, wherein said computer further 

2 comprises at least one lesson storage database, further 

3 comprising the step of storing said interactive language 

4 lesson at said at least one lesson storage database. 

1 39. The system of Claim 38 wherein said computer further 

2 comprises visual output means, further comprising the step of 

3 displaying said interactive language lesson stored in said at 

4 least one lesson storage database. 
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1 40, A method for speech recognition means to interpret 

2 student utterances of known text in a lesson comprising: 

3 receiving student utterances? 

4 accessing lesson-based speech interpretation information 

5 from one or more lesson-based database; 

6 comparing said accessed information to said receiving 

7 utterances ? and 

8 generating an output based upon said comparing. 

^ 41. The method of Claim 40 further comprising the steps of 
accessing acoustic information from at least one acoustic 

^ database and comparing said acoustic information to said 
student utterances. 

if 42. A method of providing effective speech recognition of 

P= user utterances of text comprising: 
3i inputting text to said system; 

4r generating a plurality of phoneme sequences based on said 

5 input text; 

6 generating a plurality of variations of said input text; 

7 and 

8 storing said plurality of phonen^ sequences and said 

9 plurality of variations of said input text. 

1 43. The method of Claim 42 further comprising providing at 

2 least one database having acoustic interpretation information. 
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SYSTEM AND METHOD FOR INTERACTIVE READING AND LANGUAGE 

INSTRUCTION 



ABS'PRACT OF THE IMVEMTTOW 



A reading or language interactive system which utilizes a 
mechanism to permit a computer-based instruction program in reading 
or language instruction to implement a shared reading program, 
wherein portions of the text are read by the student and portions 
by the computer itself. The proportion of the text "read" by each 
participant and the pace at which the lesson progresses can each be 
automatically altered based upon the proficiency of the student. 
In addition, the system utilizes a computer instructor to share the 
task of reading or language learning by modeling the task for the 
learner, variably controlling the amount of material that the 
student reads aloud based upon a computer assessment of the 
student's ability, helping the learner navigate through the 
instructional application, and participating in a joint performance 
that demonstrates the student's reading. 
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